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1.  Executive  Summary 


Hus  is  a  technical  Tq[K»t  of  the  results  and  lessons  learned  from  a  STARS  sponsored  technology 
transfer  demonstradtuL  The  Armament.  Munitions  and  Qionical  Command  life  Cycle  Software 
Engineeiring  Center  (AMCCOM  LCSEC)  at  tl»  Picatinny  Arsoial  was  selected  as  a  site  to 
demonstrate  that  a  STARS  siq>ported  state-of-art  software  development  process,  namely 
Cleanromn  Software  Engineering  (CSE),  could  be  successfully  tilled  in  a  ^ical  DoD 
envirmunent  Additionally,  the  denuxistration  was  viewed  as  an  opportunity  to  learn  more  about 
TechnoloQr  Transfer  in  <»^  to  support  future  efforts  at  other  DoD  Software  Support  Activities, 
sudi  as  the  one  at  Peterson  Air  Force  Base. 

This  rq)ort  covers  the  experioices  of  the  IS  month  effort  from  May  1992  through  July  1993,  but 
focuses  on  the  period  between  November  1992  and  July  1993  when  the  selected  AMCCOM 
LCSEC  projects  were  ongoing. 

Tte  Picatirmy  mission  is  accomplished  by  both  govemmrat  staff  and  by  supporting  contractor 
personnel  Hvo  dononstration  projects  were  selected:  one  was  perform^  by  government  staff 
and  one  was  principally  perfonned  by  support  contr»nots.  This  report  reflects  principally  the 
experiences  gained  from  the  goveniment  s^  project  Although  final  results  are  still  somewhat 
jnemature,  indications  are  that  diis  project  has  achieved  the  following  results: 

■  Geanroom  software  engineering  practices  and  process  guided  program 
managonent  is  a  technology  that  can  be  successfully  transfnred  to  DoD  software 
organizations.  Current  organizational  maturity  (e.g.  Software  Engineering  Institute 
Cipibility  Maturity  Model  (SEl  CMM)  Level  1)  does  not  inhibit  successful 
Geanroom  nor  process  guided  technology  transfer. 

■  Picatirmy  engineering  staff  productivity  and  quality  was  increased  while 
simultaneously  increasing  job  satisfoction  with  the  "team  oriented"  approach  of  the 
Geamoom  practices  and  process. 

■  Preliminary  findings  indicate  a  return  on  investment  of  between  2:1  and  6:1  is 
possible.  A  more  definitive  calculation  will  be  made  at  project  completion. 

This  IS  month  demonstration  project  identified  several  "lessons  learned"  for  application  to  other 
technology  transfer  efforts. 

■  Successful  technology  transfer  programs  require  five  components:  (1)  formal 
technology  training;  (2)  formal  training  in  process  guided  project  management;  (3) 
support  from  reference  handbooks;  (4)  the  use  of  a  process  support  system  (e.g. 
GBPA  and  its  successor);  and  (5)  the  availability  of  qualified  follow-on  coacl^g. 

■  Process-guided  project  management  enhances  communications  among  team 
members,  project  members  and  management 
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■  Introdacing  a  formal  engiiieeriag  proc^  such  as  Cteanroom  into  a  DoD 
CMganizaticm  vnll  require  a  significant  nmi-recurring  investment  of  time  and 
money.  The  calculation  of  the  return  on  investment  requires  establishmoit  of 
meaningful  before  and  after  metrics  of  productivity  and  quality. 

This  demonstraticMi  is  an  initial  fiilfillment  of  the  ARPA  STARS  mission  of  serving  as  a  catalyst 
for  improving  software  development  in  DoD  organizatimial  elemmts.  The  synthesis  of  the 
Picatiwny  and  STARS  efforts  are  realized  by  die  foct  dnit  the  initial  contractor  funding  fOT 
doixnistratitm  project  was  provided  by  STARS  and  Picatinny  management  has  (^ted  to  support 
the  continuatum  of  the  effort  to  fiirdier  evolve  their  entire  organization  to  use  of  the  Oeanroom 
technology  with  thdr  own  funds. 

Odier  results  include  the  fact  that  die  IBM  STARS  team  gained  actual  experience  in  sui^rting 
technohigy  transfer  from  this  effort  There  is  great  excitement  in  continuing  this  effort  in  the 
future  at  Picadnny,  and  using  the  experiences  as  a  basis  to  support  the  demonstradon  project  at 
Peterson  Air  Force  Base.  This  project  also  serves  as  additional  confirmation  that  Qeanroom 
software  engineeting  practices  are  transferraUe  and  effective. 
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2.  Tedmology  Transfer  Han  -  Overview 


The  gotl  for  the  technology  transfer  effort  fw  foe  AMCXX)M  LCSEC  at  foe  Picatinny  Ars^ial 
was  to  conduct  a  donoostratitni  of  CSE  practice  and  process-guided  project  management 
(PGPM)  at  a  DoD  Sk^tware  SiqqxMt  Center.  The  dononstration  was  to  be  facilitated  by  IBM 
and  Software  Engineering  Technology  (SET). 

AM(X!OM  LCSEC  was  selected  in  re^nse  to  their  expressed  interest  in  inqnoving  the  process 
wfajdi  they  maintain  software  in  general  and,  ^)eciftcally,  in  using  foe  CSE  technology. 
Additionally,  as  a  typical  DoD  Software  Suiqxvt  Center,  it  was  deemed  inqxvtant  to  improve  foe 
means  by  which  foe  govemmoit  spends  th^  largest  portion  of  software  money;  i.e.,  in  software 
maintraance  (as  oj^sed  to  new  software  development). 

To  conduct  foe  demonstration,  both  control  and  dononstration  groiq>s  were  idoitified.  The 
control  group  consisted  of  a  sample  set  of  ongoing  and  coiiq>letBd  software  projects  at  foe 
AMCCX)M  LCSEC.  These  projects  rqpresent  foe  use  of  "typical"  software  engineering  mefoods 
at  dw  AMCCOM  LCSEC.  Enhanconoit  projects  at  Picatinny  typically  include  foe  correction 
of  observed  problems,  the  addition  of  new  capal^ties,  and  in  some  cases,  re-engineering  of 
software.  Tbe  two  demonstration  group  projects  consisted  of  (1)  the  Mortar  Ballistics  C}onq)uter 
(MBC)  redevelopment  software  and  M2/3A1  histitatioiud  OoiKloct  of  Rre  TYainer  (I-COFT) 
softwaoe  block  update.  The  demonstration  t^spcct  of  these  projects  is  foe  adoption  of  foe  CSE 
technology  and  PGPM  techniques  as  conveyed  by  the  participation  of  IBM  and  SET.  The 
hypothesis  to  be  confirmed  or  rejected  in  this  demonstraticH)  was:  The  use  of  CSE  practices  and 
process-driven  project  mam^emoit  improves  the  effectiveness  the  AMCCOM  LCSEC 
in  its  software  suivort  mission  on  a  project  basis.  The  goal  of  any  software  development 
organization  is  to  (^elop  software,  wifoin  schedule  and  budget,  that  flawlessly  performs  its 
mission.  Of  course,  schedule  and  budget  are  foe  crnistraints  established  so  that  the  organizatimi 
contracting  for  the  software  solution  obtains  foe  software  wifo  tiie  minimum  possible  investment 

There  are  tiuee  aspects  of  an  organization’s  behavior  that  influence  how  well  they  can  achieve 
this  goal  which  are  represented  in  Hgure  1.  Hie  three  aspects  are: 

Technological  aspects  -  the  engineering  practices  that  foe  engineers  utilize  to  specify, 
develop  and  certify  foe  software  solution; 

Process  a^iects  -  foe  management  practices  applied  by  the  project  in  conducting  the 
project;  and 

Organizational  Aspects  -  foe  management  practices  foUowed  by  the  organization  and  its 
organizational  culture  which  serves  as  the  ultimate  guidepost  to  its  behavior. 

This  (tenonstration  project  was  directed  at  only  changing  foe  behavior  in  the  first  two  of  these 
three  areas  by  establishing  a  technology  transfer  program  for  the  demonstration  groups  in  a  two 
part  package  which  onphasized  both  the  original,  disciplined  focus  of  Qeanroom  and  the 
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proces»-giiided  engineoing  focus  of  the  IBM  STARS  team  effoit  Measures  were  established 
to  q>pnuse  the  intact  at  the  project  level 


Figure  1:  Three  Part  View  of  Technology  Transfer 
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In  the  demonstration  project,  no  overt  attempt  was  made  to  change  or  measure  any  change  in 
organizational  behavior  or  culture.  This  third  part  is  typically  measured  by  the  SEI CMM  which 
focuses  mi  measuring  the  organization’s  (not  project’s)  process  capability.  This  three-part  focus 
is  dqiicted  in  Hgure  1.  The  goal  was  to  establish  a  "Qeanroom  environment"  within  the 
demonstration  inoject’s  organizational  strachue.  A  "Qeanroom  environment"  exists  when  the 
objectives  and  attitudes  of  an  organization  foster  the  poroper  application  of  CSE  ideas. 

Following  our  experience  at  Picatinny  we  believe  that  if  an  organization  is  attempting  to  improve 
its  behavior  in  the  most  efficient  manner  possible,  it  should  undertake  to  upgrade  aU  three  aspects 
in  a  balanced  manner  so  that  inqirovement  in  each  area  can  feed  off  of  improvements  in  the  other 
two  aspects. 

The  technology  transfer  package  was  implemented  in  two  of  the  three  areas  as  follows: 

(1)  the  tranter  cf  Cleanroom  Engineering  practices  to  give  team  members  the  technical 
tools  that  provide  the  human  behavioral  chmiges  necessary  to  create  high  quality  software 
widi  incrmued  productivity,  and 

(2)  the  tranter  cf  process-guided  project  management  to  orient  both  individuals  and 
teams  to  thiitidng  and  working  within  a  PGPM  environment 

The  tiiird  aspect  instituting  organizational  changes  within  the  scope  of  the  two  projects,  was  not 
a  focus  of  this  initial  effort  at  Picatinny  but  is  discussed  since  it  is  a  major  aspect  of  organization 
behavior. 

The  definition  of  each  of  these  three  aspects  and  the  means  by  which  the  technology  was 
transferred  for  the  first  two  of  them  is  described  below.  The  details  of  how  the  technology 
transfer  was  implemented  is  covered  in  section  4. 
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In  ofder  to  transfer  fee  technology,  process  and  culture  for  a  Qeanioom  environment,  four 
diffeirait  tools  were  oi^loyed: 


(1)  training,  in  a  formal  classroom  setting  which  integrated  lecture  material  and  numerous 
hands-on  workshops, 

(2)  coaching,  bofe  for  project  plaiming  and  execution  as  well  as  a  medium  to  promote 
(mgoing  education, 

(3)  process  handbooks,  which  act  as  a  written  source  of  education  material  and  as  a 
reference  during  i^oject  execution,  and 

(4)  an  automated  process  support  system,  that  enforces  process  adherence  and  monitors 
task  cofiq>letion. 

Each  of  these  played  a  role  in  fee  overall  technology  transfer  and  often  were  used  together  to 
enhance  fee  effort  Table  I  summarizes  where  each  technique  was  used  wife  regard  to  each 
aspect  of  (organizational  behavior  that  effects  the  organization’s  software  engineering  maturity. 


Table  1:  Techniques  Used  in  Each  Aspect  of  the  Technology  Transfer 


Training 

Coaching 

Handbooks 

Automated 

Support 

Qeanr(X)m  Software 
Engineering  Practices 

X 

X 

X 

Process-Guided 

Project  Management 

X 

X 

X 

X 

Organization 
Management  Practices 
and  Culture 

NAS 

NAS 

NAS 

NAS 

The  sections  below  provide  a  mote  substantive  discussion  of  the  three  aspects  of  the  technology 
transfer  and  a  discussion  of  the  means  by  which  the  transfo'  was  carried  out 

Cleanroom  Software  Engineering  CSE  consists  of  a  body  of  practical  and  theoretically  sound 
engineering  principles  applied  to  fee  activity  of  software  engineering.  Qeanroom  consists  of  a 
thorough  specification  phase;  resulting  in  a  six  part  specification,  including  a  precise,  black  box 
description  of  the  software  part  of  a  systerrt  Software  development  proceeds  from  the  black  box 
specification  via  a  step-wise  refinement  procedure  using  box-structured  design  concepts.  This 
process  focuses  on  defect  prevention,  effectively  eliminating  costly  error  removal  phases  (i.e., 
debugging)  and  prcxluces  verifiably  correct  software  parts.  Development  of  software  proceeds 
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in  parallel  with  a  unge  qiedficatkm  of  the  software.  This  usage  profile  becomes  the  basis  for 
a  statfatkal  teat  of  the  software,  resulting  in  a  scientific  certification  of  the  quality  of  the 
software  part  of  the  system  so  (teveloped. 

Transfer  of  CSE  Technology  The  transfer  of  CSE  technology  was  achieved  through  formal, 
classnxnn-style  training  courses  and  follow-on  coaching  of  demonstration  team  members.  The 
courses  involved  instruction  on  die  underlying  specification,  development,  and  certification 
methods  of  CSE  and  included  in-class  workshops  so  drat  students  gained  experience  applying  the 
technology.  As  (rfien  as  possible,  workshops  woe  held  with  examples  extracted  from  the  I- 
COFT  and  MBC  projects.  Training  provided  the  introduction  to  and  initial  experience  with  the 
tools  that  would  help  enhance  individual  and  team  performance. 

Project  stqrport  was  given  to  die  team  members  of  both  demonstration  projects  through  repeated 
on-site  coaching  visits  by  CSE  experts  from  SET  and  IBM.  This  activity  helped  to  solidify  the 
new  ideas  as  team  members  saw  how  die  techniques  were  applied  to  their  sp^tific  problems. 

The  major  intent  of  the  training  and  coaching  was  to  establish  the  human  behavioral  changes 
necessary  to  develop  better  software.  Implementing  CSE  is  an  intellectually  challenging  process 
that  instills  specific  values  into  its  participants.  For  example,  the  focus  on  product  quality,  a 
major  Qeanroom  theme,  instills  a  "get  it  right  the  first  time"  attitude  into  the  members  of  CSE 
teams.  As  successes  are  made  and  nndlestones  conquoed,  new  CSE  teams  often  report  significant 
inqirovements  in  job  satisfaction,  team  spirit,  and  the  desire  to  continue  quality  improvements. 
A  significant  focus  of  the  coaching  effort  was  to  positively  reinforce  each  project  success  in  order 
to  create  a  stronger  identity  with  the  project 

Such  behavioral  changes  within  a  project  are  improved  by  active  participation  from  all  levels  of 
the  organizational  hierarchy  from  contributing  technical  leads  to  engineering  management  An 
iititial  plan  was  for  the  project  staffs  to  work  closely  as  teams,  rather  than  as  individuals. 
Additionally,  die  intention  was  for  the  staffs  to  be  motivated  and  excited  about  what  they  were 
doing;  that  is,  have  a  strong  identity  with  the  process  and  project  Thus,  coaching  contained  a 
"cheerleading"  aspect  designed  to  create  a  heathy  Qeanroom  environment 

Reinforcement  of  CSE  was  provided  through  the  availability  of  a  six  volume  set  of  process 
manuals  to  the  demonstration  groups.  These  process  manuals  were  an  integral  part  of  the  training 
program  and  were  discussed  in  detail,  both  during  the  formal  training  sessions  and  off-line  as  a 
part  of  the  follow-on  coaching  activities.  Their  purpose  was  to  augment  the  training  by  providing 
refooice  information  to  AMCCOM  LCSEC  engineers  using  Qeanroom  concepts.  Ihey  serve 
as  a  single  reference  source  for  resolving  questions  about  specific  issues  concerning  process 
adherence. 
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The  process  manuals  are  organized  as  follows: 

Volume  1:  Geanroom  Engineering  Process  Introduction  and  Overview 

Volume  2:  Organization  and  Project  Formation  in  the  Qeantoom  &ivironment 

Volume  3:  Project  Executitm  in  die  Cleanroom  Environment 

Volume  4:  Specification  Team  Practices 

Volume  5:  Development  Team  Practices 

Volume  6:  Certification  Team  Practices 

The  division  of  the  volumes  represents  a  separation  of  concerns  for  the  various  project 
stakdiolders. 

Process-Guitted  Project  Management  CSE  takes  place  within  a  formal  process  that  clearly 
defines  the  tasks  necessary  for  the  engineering  effort  to  progress,  the  completion  conditions  for 
each  ta^  and  the  control  flow  that  dictates  the  order  of  work  on  each  task.  Process-guided 
project  management  entails  the  use  of  a  clearly  defined  process  as  the  approach  to  be  used  to 
complete  the  particular  project  The  intent  with  process-guided  project  management  is  to  give 
oigineers  a  clear  and  u^erstandable  roadmap  which  they  can  follow  and  by  which  diey  may 
track  progress  towards  project  conviction. 

Transfer  of  PGPM  methods  Awareness  of  software  process  is  a  key  issue  in  successfully 
transferring  technology  to  an  organization  and  to  an  organization’s  long  term  success  with 
applying  CSE.  The  project  staffs  at  AMCCOM  LCSEC  received  an  introduction  to  process 
ddinition  and  process  guided  engineering  in  the  context  of  CSE.  Coaching  also  reinforced  the 
invortance  of  following  the  defined  process  and  using  the  process  definition,  which  defines  all 
of  the  possible  project  alternatives,  to  support  the  selection  of  correct  project  choices. 

In  addition  to  training  and  coaching,  the  engineering  handbooks  provided  a  key  reinforcement 
of  the  concepts  of  process-guided  engmeeting.  Each  volume  defines  the  tasks  and  the  control 
flow  between  the  tasks  necessary  to  conduct  the  specific  process  which  is  the  focus  of  the 
manual  Engineering  processes  are  defined  as  formal  control-flow  procedures  with  specific 
completion  conditions.  Collections  of  engineering  processes  also  have  the  same  level  of 
fonnalized  control  flow  and  conviction  conditions.  Thus,  each  engineer,  manager  or  other  staff 
memba  has  well  defin^l  roles  and  tasks  that  exist  as  a  part  of  a  larger  software  process. 

The  application  of  the  process  is  supported  by  formal  enactment  of  the  tasks  defined  in  the 
handb^k.  For  the  MBC  team,  this  enactment  is  automated  in  the  Cleanroom  Engineering 
Process  Assistant  (C£PA),  an  automated  process  support  system  which  has  the  following  mission; 

1.  To  minimize  realization  productivity  losses,  which  is  to  reduce  the  time  lost 
because  supporting  activities  are  not  properly  coordinated.  CEPA  will 
significantly  improve  the  probability  that  all  of  tiie  pre-requisites,  tools  and  data 
that  an  engineer  needs  to  do  a  task  are  available  with  no  wasted  time  on  his  or  her 
part 

2.  To  enable  engineers  to  follow  the  Cleanroom  process  and  thereby  obtain  all  of  its 
boiefits. 


ID52  -  Bnal  Evaluation  Report 


CDRL 05503-004  Page? 


3.  To  mforce  the  Cleanroom  process  in  die  tiK)St  unobtrusive  way  possible  by  being 
user-Mendly. 

4.  To  enable  all  levels  of  management  to  plan«  schedule  and  control  project  tasks  and 
to  ensure  that  tlu  required  reviews  and  verifications  take  place. 

5.  To  facilitate  the  coU^on  of  all  required  metrics  for  providing  statistical  control 
of  the  process  and  for  providing  better  estimates  of  d^elopment  time  and  cost 

6.  To  update  on-line  state  data,  the  data  needed  to  develop  the  product  and  make  it 
immediately  available  to  aU  membm  of  the  project  team. 

7.  To  improve  formal  and  informal  communication  between  the  members  of  the 
group. 

The  I-COFT  team  was  to  practice  forms-based  enactment  as  opposed  to  the  automated  enactment 
using  CEP  A,  so  tiiat  comparisons  could  be  made  about  the  level  of  benefit  achieved  from  the 
tool. 

The  engineering  handbooks,  and  the  two  types  of  oiactment  give  project  staff  a  way  to  use  a 
project  framework  (the  process  model  for  a  project)  that  facilitates  scheduling,  task  dispatching 
and  task  statusing. 

Organizatimial  Changes  The  goal  of  the  demonstration  project  is  not  to  define  a  complete 
organizational  assessment  model  for  software  engineering.  In  fact,  die  SEI  CMM  adequately 
perfrums  this  for  us.  The  AMCCOM  LCSEC  organization  has  undergone  a  software  process 
assessment,  as  defined  by  the  SEI,  and  was  assessed  to  be  a  level  1  site.  The  goal  of  the  initial 
effort  is  to  promote,  witiiin  the  two  demonstration  projects,  entiiusiasm  for  the  Cleanroom 
engineering  practices  and  the  motivation  to  develop  high  quality  software  products.  The 
organizational  aspects  were  not  a  part  of  the  technology  transfer,  since  the  focus  of  the 
demonstration  was  to  support  two  projects,  not  to  change  tibe  organization. 
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3.  Overview  of  AMCCOM  LCSEC  Organization  for  the  Demonstration  Project* 

AMCXXDM  LCSEC  is  an  SEI  CMM  level  1  software  support  site  v^thin  the  DoD  that  performs 
software  enhancement  tasks  for  a  set  of  Army  weapons  systems.  Much  of  the  enhancement 
activity  is  performed  by  contractors  with  government  ovwsight  The  major  product  types 
maintained  at  Picatinny  are  Ere  control  systems  and  training  devices.  The  maintenance  projects 
for  these  products  are  typically  software  block  updates  (SBU).  SBU’s  are  an  accumulation  of 
change  requests  from  the  customer,  to  be  delivered  normally  within  12  to  18  months. 

The  (tesire  for  CSE  technology  was  a  result  of  the  recognition  by  AMCCOM  LCSEC 
management  that  the  software  process  was  not  under  intellectual  control.  Each  new  software 
project,  whether  performed  by  contractors  or  civil  servants,  was  treated  largely  as  new  activity 
that  did  not  necessarily  draw  on  prior  experience  for  process  improvement  The  only  factor  that 
perpetuated  experience  was  people,  be  it  government  or  contractor,  who  participated  in  the  same 
projects  time  after  time.  Documentation  received  by  Picatinny,  when  they  were  given  systems 
to  maintain,  was  poor  and  no  defined  process  existed  for  maintaining  continual  project  control. 
In  other  words,  die  state-of-the-practice  consisted  of  classic  craft-based  software  engineering 
practices  that  are  ad-hoc  in  nature,  as  opposed  to  disciplined  software  engineering  practices.  To 
their  credit,  this  was  recognized  by  the  AMCCOM  LCSEC  management  and  was  the  basis  for 
their  move  to  enhance  their  software  engineering  capabilities. 

Compounding  the  problem  of  process  immaturity  at  the  AMCCOM  LCSEC  is  the  lack  of  a 
form^  task-oriented  planning  and/or  schedule  adherence.  Among  project  staff,  formal  scheduling 
and  schedule  adherence  are  not  emphasized,  only  that  activity  on  a  specific  project  intensifies  as 
deadlines  come  close  and  deliverables  are  imminent  This  is  an  attribute  of  sta^  members  being 
spread  across  a  number  of  projects,  with  work  being  driven  by  the  most  pressing  deadline.  In 
this  situation,  it  is  difficult  for  engineers  or  team  leaders  to  set  up  a  well  defined  plan  or  schedule 
for  solving  problems. 

The  combination  of  an  undefined  manner  of  doing  work,  along  with  a  lack  of  task-oriented 
scheduling  created  somewhat  of  a  morale  problem  among  software  engineers  at  the  AMCCOM 
LCSEC.  They  did  their  work  well  because  of  individual  skills,  but  often  seemed  to  be  stuck  in 
the  same  "groove,"  where  the  same  situations,  in  terms  of  schedule,  would  arise  year  after  year. 
A  general  lack  of  enthusiasm  pervaded  our  initial  discussions  with  project  teams. 

Despite  these  difficulties,  however,  the  customers  (various  users  within  the  US  Army)  indicate 
that  they  are  basically  content  with  the  quality  of  the  products.  This  really  is  a  testament  to  the 
skill  of  staff  at  the  LCSEC,  where,  despite  working  as  a  typical  DoD  Software  Support  Activity 
(SSA),  they  have  provided  quality  work.  Not  many  field  reports  of  failures  are  submitted  by 
their  customers,  due  to  extensive,  pre-release  usage  testing.  Unfortunately,  evidence  suggests  that 


1 


This  overview  pertains  only  to  tbe  demonstration  projects  and  not  to  tbe  Picatinny  organization  as  a  whole. 
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tilts  may  be  a  result  of  the  absence  of  formal  failure  observation  and  reporting  mechanisms, 
making  die  field  quality  of  AMCCOM  LCSEC  developed  products  difficult  to  ascertain. 

AMCCOM  LCSEC  management  recognized  the  problems  with  their  state  of  the  practice  and  took 
the  midative  to  recognize  CSE  and  PGPM  as  the  mechanisms  with  which  to  facilitate  the  desired 
cultural,  technical  and  process  changes. 

The  control  groiqis  represoit  the  state-of-the-practice  at  die  AMCCOM  LCSEC.  Baseline  metrics 
were  collected  in  cnder  to  gain  insight  into  project  practices  and  to  establish  a  basis  of 
conqiaiison  to  die  demonstration  Oeamoom  groups.  Table  n  presents  the  baseline  metrics  for 
die  control  groiqi.  Formulas  for  die  measures  are  defined  in  Appendix  A.  These  nwtrics  are 
presented  with  the  caution  that  some  data  collection  mechaninns  are  unreliable;  resulting  in 
inaccuracies.  Specifically,  the  failure  rates  of  the  control  group’s  released  software  were  difficult 
to  find  or  even  non-existent,  which  may  be  a  result  of  not  having  a  formal  process  by  which  the 
Army  users  provide  feedback  to  Picatinny.  Pre-Deployment  failures  are  typically  not  collected 
in  this  organization.  The  numbers  in  Table  n  are  similar  to  results  report  by  Mosemann  for 
other  projects  within  the  DoD  fAda  and  C-H-:  A  Business  Caiift  Analysis.  July  1991]. 


TaMe  II;  Baseline  Metrics  for  Control  Group  Projects 


PROJECT  /  MEASURE 

^  Contnrf  Group 
Projects 

Only  Government 
staffed  Control  Group 
Projects 

Technical  staff  months 

192 

135 

KLOC  (*) 

23.14 

12.32 

Pre-Dqiloyment  Failure  Corrections 

N/A 

N/A 

Pdst-Deployment  Failure  Corrections 

N/A 

N/A 

DERIVED  METRICS 

PRODUCnvrrY  -  LOC/Statr  Month 

121 

91 

PRE-QUALITY  -  Failure/KLOC 

N/A 

N/A 

POST-QUALFTY  -  FaOure/KLOC 

N/A 

N/A 

7^^E^^o!^ute^!sSn^RSS!^5a33ara3eveIopSr!!n^o^o3^ormuIa^ 


( New  Lines  of  Code  +  0.2  *  Modified  Lines  of  Code )  /  1000 
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4.  Technology  Transfer/Project  Implementation 

The  1D52  task  began  in  late  April  1992  and  continued  until  July  31.  1993.  The  Schedule  of 
Activities  (Hgure  2)  on  the  next  page  provides  an  outline  of  the  chronological  relationships 
between  various  task  activities.  The  activities  include  boA  the  technology  transfer  efforts,  and 
some  schedule  information  for  the  demonstration  projects. 

Beginning  in  April  1992,  monthly  meetings  were  instituted  with  AMCX^OM  LCSEC  management 
to  discuss  die  transfer  of  CSE  practioes  and  PGPM.  Meetings  consisted  of  preparing  for  the 
technology  transfer,  introducing  AMCCXJM IXIISEC  management  to  CSE,  and  conq)leting  tasks 
to  facilitate  die  technology  transfer.  During  these  meetings  die  two  demonstration  projects  were 
identified. 

In  parallel  with  these  meetings,  engineering  luuidbooks  were  prqiared  and  CEPA  (Qeanroom 
Engineering  Process  Assistant)  was  enhanced.  Other  tasks  included  the  preparation  of  a 
statement  of  work  which  integrated  required  Qeanroom  process  aspects,  as  well  as  descriptions 
of  d»  enhancemrats  to  be  implemented,  into  a  contractor  task  order.  On  August  21, 1992,  John 
Foreman,  Director  of  the  STARS  program  participated  in  a  meeting  to  discuss  the  interest  of 
STARS  in  die  context  of  the  goals  of  the  Picatinny  initiative.  At  this  time,  the  schedule  for  the 
dononstration  was  established. 

The  two  projects  selected  for  the  demonstration  of  CSE  were  the  M2/3A1 1-COFT  and  the  MBC 
redevelopment  The  M2/3A1  (or  Bradley  Fighting  Vehicle)  I-COFT  represents  a  software  block 
update  for  an  application  of  approximately  200  KLOC  Fortran  code  which  controls  a 
gurmer/commander  trainer  for  the  Bradley  Fitting  Vehicle.  The  MBC  redevelopment  will  re¬ 
engineer  existing  mortar  ballistics  software  using  the  Ada  programming  language  to  run  on  a 
number  of  host  machine.  The  I-COFT  effort  is  contracted,  with  effort  monitored  by  AMCCOM 
LCSEC  staff  and  die  MBC  is  being  redeveloped  by  government  employees.  Both  project  teams 
received  training  and  engineering  handbooks  as  well  as  proactive  and  reactive  coaching,  llie  I- 
COFT  team  was  to  practice  forms-based  enactment,  as  opposed  to  the  automated  enactment  of 
CEPA,  so  that  conqiarisons  could  be  made  to  the  level  of  benefit  achieved  from  the  tool.  The 
forms-based  enactment  approach  allowed  the  l-COFT  team  to  use  process-guided  management 
principles  in  planning  and  delegation  of  project  tasks. 
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Figure  2:  Schedule  of  Activities 
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ARPA  Project  Support  iPicatinny  Project  Support 


The  training  program  was  (Organized  into  2  courses,  each  of  five  days  duration  covering 
fvocess/specification  aiui  develojnnent^certification,  respectively.  The  scl^ule  for  the  training 
courses  was  as  follows  in  Table  m*. 

Table  m:  Sdiedule  of  Training 


MBC  PROJECT 

I-COFT  PROJECT 

iMov.  2-6,  1992: 

Trauung 

Dec.  6-10,  1992: 

Training 

Feb.  22-26,  1993: 

Contractor 

March  8-12,  1993: 

Certification  team 
repeat  session 

Contractor/Subcontractor 

March  17-19,  1993: 

Subcontractor 

March  25,  1993: 

Subcontractor 

Perh^ts  die  most  effective  means  of  technology  transfer  is  coaching,  via  both  on-site  visits  and 
telephone  assistance  from  qualified  consultants.  When  expert  engineers  exist  in  software 
engineering  sites,  they  are  used  as  coaches  for  others  who  are  learning  the  new  methods. 
However,  when  discqilined  software  engineering  is  not  being  practiced,  as  is  the  case  with 
Picatinny,  the  only  place  to  look  for  qualified  coaching  is  outside. 

This  aspect  of  the  technology  transfer  was  put  into  effect  with  personnel  from  both  SET  and  IBM 
at  a  level  of  effort  of  tqiproximately  2  days  per  week.  The  effort  began  in  November  1992,  and 
continued  until  the  end  of  the  ta^  A^tionally,  staff  were  always  available  for  telephone 
support,  when  a  member  of  IBM  and  SET  staff  was  not  present  A  number  of  otho:  aspects  of 
project  support  also  occurred,  which  represent  a  rignificant  portion  of  the  IBM  and  SET  effort 
as  th^  formed  the  basis  for  the  technology  transfer. 
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Mfiy  \m: 
Nov  1992: 

Jan  1993: 

Fdb  1993: 


Table  IV:  Other  Significant  Technology  Transfer  Activities 

_ MBC  PROJECT _ I-COFT  PROJECT 

bxaiiq>le  SOW  for  contractor 
using  Qeannxmi  delivered 

Engineering  handbooks  delivered  Initial  Meeting  w/  I>C(^ 
CEPA  delivered  Q)ntractiur  in  Boston 

Suiqxwted  software 
develoimient  plan 
prqparaticni  by  contractor 

Manual  ooactment  mechanism 
delivered  to  contractor 
Engineering  handbooks 
delivend  to  contractor 
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5.  Observations 


The  fdlowing  obsovations  aie  a  compilation  of  SET  and  IBM  experioices  with  the  MBC  and 
I-CXXT  teams.  These  observatimis  are  in  the  contmct  of  SET’s  and  IBM’s  other  experiences  with 
repladiig  craft-based  practices  with  engineering-based  practices,  both  in  the  private  sector  and 
with  govenunent  employees.  One  must  keep  in  mind  these  observatums  are  preliminary  since 
the  projects  have  not  been  ccai^leted.  These  observatitMis  are  firmer  relative  to  the  MBC  project 
team  since  diat  (nroject  is  die  faidiest  advanced. 

1.  The  assigned  project  teams  were  able  to  assimilate  and  even  adapt  the  Cleanroom  Software 
Engineering  practices. 

A  common  v^nry  among  managers  when  hearing  about  Cleanroom  is  that  it  is  too  hard  or  too 
madiematical  for  their  staff.  At  Picatinny,  engineers  were  able  to  apply  and  adapt  the  Cleanroom 
pracdoes  to  the  needs  of  their  jnoject 

Discqdined  en^neering  in  a  team  environmmt  requires  rigor,  cooperation  of  individuals,  and  the 
creativity  to  iqpply  diemy  to  real  world  proUems.  This  creates  a  challenging  work  environment 
Aat  ten^  to  bring  out  die  best  in  both  individuals  and  teams. 

A  prime  exanqite  of  the  accomplishmaits  of  die  MBC  team  was  the  tailoring  of  the  box 
structures  algoridun  to  meet  bodi  their  qiplication  environment  and  the  target  programming 
language  Ada.  MBC  team  members  have  matte  original  contributions  to  the  expression  of  box 
structure  constructs  u^g  the  Ada  language,  which  will  have  applicability  across  many 
Cteanrocun  projects.  This  has  benefited  both  die  project,  in  terms  of  constructive  methods,  and 
die  individoal  teun  members,  in  terms  of  a  sense  of  accomplishment  The  team  has  enjoyed 
using  die  various  Cleanroom  techniques  and  have  seen  many  real  accomplishments.  The 
specification  team  is  ctuivinced  that  diis  is  the  most  complete  and  precise  pecification  diey  have 
evet  written.  The  step-wise  refinement  and  verificatimi,  which  drives  enginears  to  define  a  small 
stqi  to  take  at  a  time,  take  that  step,  and  thoi  confirm  its  correctness,  has  also  bera  successful. 
The  devdqmiait  team  is  convin^  that  they  have  a  great  design  and  have  minimized  the 
amount  d  code  they  need  to  develop. 

Pnrdietmore,  as  die  MBC  team  have  almost  completed  thdr  first  increment,  they  have  already 
shown  major  gains  in  productivity.  Early  estimates  show  that  productivity  has  doubled  despite 
die  learning  curve  of  working  with  a  new  mediodology.  Moreover,  this  measure  includes  time 
pent  toward  an  entire  imoduct  specification,  which  will  make  future  increments  less  time 
assuming.  Thus,  team  members  are  optimistic  about  continued  increases  in  their  productivity 
(aldiough  future  predictions  can  only  be  assertions  and  remain  to  be  confirmed  at  project 
completion). 
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2.  Suffmimde  has  inq»"oved  on  the  project  teams. 


Anotfier  comnon  fear  of  managers  when  bearing  about  Cleanroom  is  that  their  staffs  will  not  like 
it  due  to  die  rigor  oi  the  process  and  the  absence  of  positive  feedback  through  debugging.  It  has 
bem  our  eiqperienoe  at  oAia  places  where  we  have  introduced  Cleanroom.  Picatinny  is  no 
excqitimi  in  diat  whoi  an  organization  rqilaces  craft-based  practices  with  engineering-based 
practices,  morale  improves.  The  reason  seems  to  be  that  craft-based  practices  do  not  result  in 
a  high  quality  product  When  oigineers  leam  to  use  the  Qeanroom  practices,  they  know  they 
can  do  die  quality  job  they  have  been  striving  to  achieve. 

At  die  AMCCOM  LCSBC  all  the  engineers,  both  in  informal  contacts  and  in  a  quesdoniuure 
distributed  to  die  engineering  staff,  reported  morale  inqirovements.  The  AMCCOM  LCSEC 
management  has  also  confirmed  the  existence  of  the  improved  morale  and,  of  course,  is  favorably 
inqnessed. 

3.  The  application  of  Cleanroom  Software  Engineering  practices  and  process-guided  project 
management  for  this  project  were  under  the  intellectual  control  of  the  engineering  stcff. 

As  with  any  new  approach,  the  intellectual  control  of  the  technologies  and  process  reside  with 
the  trainers/coaches.  As  the  MBC  project  started,  the  transfer  of  the  intellectual  control  did  not 
shift  to  the  govemment  staff.  This  was  primarily  the  result  of  a  relatively  low  level  of  effort  on 
this  project,  due  to  time  commitments  on  other  projects.  During  this  time  the  coaching  effort  was 
used  primarily  to  keep  die  process  and  practices  under  intellectual  control  for  the  engineering 
staff,  since  they  did  not  have  time  to  "make  die  project  dirir  own."  Once  the  level  of  effort 
increased  for  this  project,  die  government  sraff  had  the  processes^racdces  and  the  project 
completely  under  ^ir  intellectual  control,  with  the  coaching  providing  technical  support  As 
mentioned  above,  AMCCOM  LCSEC  staff  extencted  the  practices  of  the  concepts  to  solve  their 
specific  proUerit  They  "made  the  project  their  own"  when  they  applied  die  process  and  practices 
in  the  manner  they  saw  correct  while  consistent  with  the  principles,  to  solve  their  particular 
IHcject  probl«ns.  A  major  lesson  is  that  projects  must  immediately  start  using  the  new 
qiproaches  aggressively  in  order  to  gain  intellectual  control  over  the  new  approaches  and  the 
project  itself. 

4.  Communication  among  teams  (and  between  team  members)  is  greatly  enhanced  through 
process-guided  project  management. 

An  inqiortant  ingredirot  of  any  process-guided  activity  is  communication  of  contributing  teams 
and  in^viduals.  One  aspect  of  this  was  that  no  team  culture  existed  at  the  AMCCOM  LCSEC; 
meaning  that  no  real  notion  existed  of  how  teams  are  supposed  to  behave  during  project 
execution.  This  problem  manifested  itself  in  many  different  ways.  Testing  teams  often  did  not 
receive  specification  updates  (and  failed  to  ask  for  them).  Also,  work  tended  to  be  duplicated 
by  multiple  team  members  because  the  division  of  tasks  was  unclear  and  communication  between 
members  occurred  too  seldont 
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There  were  two  aspects  of  solving  this  problem  at  Picatinny.  The  first  was  to  establish  effective 
commonication  among  team  members  and  the  second  was  to  estaUish  communication  among  the 
diffierent  dqMurtments  involved  in  the  project  Our  observation  indicates  that  communication 
ammig  team  members  significantly  inqnoved  via  the  team  a{^noach  and  strengthened  through  the 
use  of  CEPA.  Team  members  rqrort  that  they  readily  use  each  other  as  information  sources, 
quality  dieclcs,  etc.  Team  reviews  are  effective  and  informative.  However,  the  second  a^>ect, 
cmmmmicatitm  between  dq>artments,  continues  to  be  a  problon.  The  MBC  certification  team 
members  wmk  for  a  different  dquotment  than  the  q)ecification  and  develofnnent  teams. 
Resulting  proUems  are  that  the  certification  team  finds  themselves  working  ^m  outdated 
q>ecificati(»is.  Furthemxare,  the  certificatitm  team  seam  to  duplicate  each  other’s  work.  A 
fittore  goal  is  to  be  aUe  to  dtq;>licate  the  success  of  the  iq)ecification  and  development  teams  in 
the  certification  team,  primarily  by  itrqnoving  communicatiom.  A  mote  concerted  effort  should 
have  been  made  the  coachm  to  minimize  tiiese  communicatiom  probleim. 

5.  The  team-oriented  approttch  of  CSE  saw  immediate  acceptance  and  realized  both  tangible  and 
intangible  ben^ts. 

A  key  ingredioit  of  Qeamoom  is  that  a  team  amplifies  human  performance.  The  smq>le  idea 
that  many  mmds  are  better  than  one  makes  the  outlook  fm  quality  good.  However,  some  less 
tangible  benefits  were  realized  as  welL  The  fact  that  the  entire  team  is  re^nsible  for  quality, 
in  a  series  of  checks  and  reviews,  puts  pressure  on  the  team  and  not  on  individuals.  This 
pressure  creates  a  rdiarKe  on  team  activity  over  individual  performance.  Furtfaernmre,  as 
successes  are  encountered,  the  entire  team  takes  credit,  not  a  single  individual;  thus,  cementing 
the  teamwork  concepts.  The  bottom  line  is  tiiat  teamwork  inqnoves  individual  performance. 

Our  observation  is  that  die  MBC  team  now  works  within  an  effective  team-oriented  environment 
We  believe  that  furdier  use  of  Oeanroom  will  establish  a  strong  ream  mentality  that  will  save 
to  further  inqnove  tiie  initial  good  results. 

6.  Coaching  is  a  key  ingredient  of  technology  tranter  success. 

Altiioagh  die  training  was  rigorous  with  a  mixture  of  tiiemy  and  hands-on  workshops,  students 
leam  at  different  rates.  Coaching  allowed  SET  and  IBM  st^  to  re-educate  the  slower-to-adopt 
project  staff  mendiers  and  keep  the  entire  team  on  a  common  level  of  knowledge  and  expertise. 
IBM  and  SET  technical  jnesmce  at  project  inception  and  during  project  execution  helped  solidify 
the  transfer  of  die  technology  and  ensured  t^t  the  project  got  started  in  the  most  efficient 
manna. 


Furthermme,  there  was  a  gap  betweoi  the  end  of  training  and  the  start  of  the  project  and  some 
of  the  education  was  forgotten.  Coaching  became  die  mechanism  to  re-educate  and  supplement 
the  original  training.  Furdier,  as  good  ideas  were  conceived  by  some  ream  members,  it  was 
possible  to  see  that  all  membos  were  supplied  with  the  new  ideas. 
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As  die  project  progressed,  die  CSE  ideas  needed  to  be  adapted  to  the  specific  Picatinny 
aivinxiineat  Com^hes  were  used  to  discuss  (tesign  alternatives  and  to  help  in  refining  the 
technology  to  best  serve  the  plication. 

PeriuqM  the  most  unnoticed  but  effective  use  of  coaching  was  in  the  positive  reinforcement  the 
CSE  trainets  wero  aUe  to  give  to  the  team  members  and  the  team  as  a  whole.  Coaches  are 
recognized  as  erqmrts.  When  experts  comment  positively  on  original  ideas  by  a  team  member, 
die  effect  can  be  enormous  in  terms  of  self-esteem  and  sense  of  accomplishmrat  and  contribution. 
The  CSE  trainers  tried  to  positively  reinforce  those  making  such  contributions  and  encourage 
others  to  seek  answers  beyond  the  limits  of  current  knowledge.  The  "cheerleading"  approach 
increased  project  satisfaction,  which  motivated  greater  imiject  performance. 

The  idea  of  coaching  with  positive  reinforconent  was  Erst  formally  tried  out  by  IBM  and  SET 
cm  the  Picatinny  project  ba^  on  the  hypothesis  diat  it  would  be  helpful  in  technology  transfer. 
The  realized  benefits  far  exceeded  our  expectations.  Based  on  this  experience,  it  is  now  believed 
that  coaching  should  be  a  formal  part  of  any  technology  transfer  effort 

7.  Process-guided  project  management  supports  engineers  in  mastering  a  new  technology. 

Process-driven,  now  referred  to  as  "Process-guided,"  project  management  is  one  of  the  two  basic 
technologies  being  advanced  by  the  STARS  program.  The  Picatinny  project  was  the  first  project 
on  which  dus  key  idea  has  been  employed.  The  MBC  project  was  to  utilize  an  automated  system 
for  suf^rting  project  process  enactment  and  the  CX)FT  project  was  to  utilize  manual  process 
enactment 

It  was  obsoved  that  automated  process  support  is  quite  helpful  in  supporting  technology  transfer. 
This  is  in  spite  of  some  of  the  shortcomings  of  the  system  that  the  MBC  staff  was  asked  to  use. 
The  developers  of  CEPA  learned  a  great  deal  almut  how  people  use  such  a  system;  and 
consequoidy,  requirements  for  an  enhanced  process  support  system  were  identified  and  modified. 
The  autonutied  process  support  system  that  is  to  be  transferred  to  Peterson  Air  Force  Base  has 
been  inqiroved  as  a  result  of  this  usage. 

The  reason  an  automated  process  support  system  seems  to  support  technology  transfer  can  be 
summarized  as  follows.  When  doing  something  for  die  first  time,  one  often  asks,  "What  do  I  do 
next?"  or  "When  will  I  be  done?"  This  indicates  a  lack  of  understanding  the  big  picture,  where 
engineers  can  clearly  place  their  efforts  in  a  project  context  This  is  not  only  an  attribute  of  first 
time  usage  of  techniques  or  a  process,  but  also  an  indication  that  a  clearly  defined  process  does 
not  exist  or  is  not  effectively  managed. 

By  placing  die  Qeanroom  techniques  within  a  fully  defined  process,  AMCCOM  LCSEC 
engineers  knew  precisely  what  step  they  were  currently  on,  as  well  as  what  had  been  completed 
and  what  remai^  to  te  done.  Giving  each  individual  the  foresight  that  showed  where  they 
were  in  the  context  of  the  entire  project  strengthened  project  identity  and  boosted  morale. 


IDS2  -  Hnal  Evaluation  Report 


CDRL  05503-004  Page  18 


The  results  of  this  project  also  indicate  that  experienced  engineers  will  also  gain  producdvity 
boiefits  by  eiiq>loying  a  process  support  system  but  that  can  only  be  tested  by  comparing  the 
performance  of  experienced  teams  which  was  not  possible  at  Picatinny. 

8.  CEP  A,  the  Cleanroom  Engineering  Process  Assistant,  despite  some  shortcomings,  provided 
vaiutdtle  process-guidance  support  for  the  project. 

There  were  a  number  of  known,  as  well  as  discovered,  shortcomings  in  developing  and  using 
CEPA.  It  was  an  enhancenomt  of  a  prototype  systrai  first  developed  during  the  STARS  S  Phase. 
The  enhanced  system  was  to  provide  support  to  engineers  using  a  specific  Qeanroom  process 
modeL  This  approach  was  known  to  be  somewhat  limiting,  but  was  used  in  order  to  determine 
the  level  of  cemstraint  necessary  for  engineers  to  easily  adopt  process  guided  engineering. 
Although  die  en^neers  (fid  report  finding  the  product  constraining,  CEPA  did  allow  engineers 
to  idoitify  tlw  tasks  assigned  to  them  and  locate  all  files  necessary  to  con^lete  the  tasks.  Team 
leaders  could  also  focus  their  management  effort  based  on  assigned/outstanding  and  completed 
tasks.  This  status  reporting  feature  allowed  team  leaders  to  manage  project  tasks  at  a  more 
reas(»iaUe  level  of  granularity,  which  permitted  them  to  maintain  the  projea  under  greater 
intellectuai  control. 

CEPA  was  viewed  as  being  tightly  coupled  widi  the  process.  As  a  result,  formal  training  in 
using  it  was  not  ^ven,  which  would  have  also  made  its  use  more  effective.  The  lack  of  CEPA 
training  was  a  significant  shortcoming  that  needs  to  be  rectified  in  future  technology  transfer 
efforts.  Additionally,  formal  training  in  using  the  underlying  tools  in  CEPA  would  have  been 
useful  Other  problems  widi  the  CEPA  iii^lementation  u^  at  IHcatinny  included  a  clumsy  user 
interface  and  difficulties  in  using  the  software  on  a  network. 

The  con^>arison  between  manual  and  automated  oiactment  for  COPT  and  MBC  respectively, 
which  was  an  intent  of  the  experimrat  was  also  unsuccessfuL  The  reason  for  this  was  the 
significant  difference  between  process  models  for  the  two  projects.  On  the  other  hand,  the  fact 
that  task  statuses  were  automatically  gathered  fen  MBC,  through  CEPA,  was  an  added 
convenioice. 

9.  The  insertion  of  process  and  technology  aspects  into  a  Statement  of  Work  are  critical  in 
having  contractors  use  a  specified  set  of  ideas  to  solve  problems. 

Aspects  of  Cleanroom  software  engineering  and  process-guided  project  management  were  inserted 
into  the  statement  of  work.  These  aspects  defined  the  content  of  some  of  the  standard 
deliverables,  which  the  contractor  would  prepare  for  the  government  In  addition,  the  Statement 
of  Work  communicated  the  means  of  support  the  contractor  would  have  in  order  to  acquire  the 
technologies.  Had  these  aspects  not  been  stated  in  the  Statement  of  Work,  there  is  little 
probability  that  the  contractor  would  have  used  the  Cleanroom  concepts.  The  obvious  reason  is 
that  no  one  is  "forcing"  them  to  use  the  ideas.  Contractors  are  typically  given  a  product  to  build, 
the  definition  of  a  process  to  follow  to  build  the  product  is  a  new  perspective.  If  it  is  not  in  the 
Statement  of  Worl^  it  becomes  a  negotiable  item. 
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10.  Specific  technologiceU  aspects  of  the  Cleanroom  software  engineering  practices  were  easily 
and  successfully  used. 

Using  s^)ecific  techniques  are  means  by  which  engineers  change  their  betutvior  and  improve  their 
performance.  Three  techniques  in  specific  were  discussed  by  project  staff  as  being  major  sources 
of  their  inqnoved  petfonnance.  Ttese  techniques  are  team  reviews,  Qeanroom  specifications 
and  box  structured  design,  and  are  described  in  greater  detail  below: 

Team  reviews,  although  experiencing  a  slow,  awkward  ^art,  were  cited  by  team 
members  as  one  of  the  most  successful  aspects  of  the  new  activity.  Members 
report  that  the  team  responsibiiity/credit  eakd  misgivings  about  participating  in 
such  a  big  project  This  negated  "finger  pointing”  that  existed  in  previous  projects 
and  allowed  even  difficult  personality  combinations  to  work  together.  The  result 
was  that  everyone  participated  and  worked  as  a  team  toward  project  success  and 
completion.  Morale  increased  sharply  as  groups  of  individuals  transformed  into 
an  effective  software  team. 

Cleanroom  specification,  most  notably  black  box  documentation,  was  cited  as 
being  respon^le  for  gains  in  productivity.  Many  talented  engineers  existed  on 
the  project  and  their  productivity  was  significantly  enhanced  when  working  from 
a  well  defined  problem  statement  The  completeness  of  the  specification  was  the 
main  reason  cited  for  the  team’s  confidence  that  they  were  producing  a  high 
quality  product 

Box  structured  design  is  credited  with  focusing  the  code  generation  process  and 
with  making  team  reviews  more  effective.  The  team  enjoyed  the  orderly  process 
of  developing  software.  It  got  than  started  more  quickly  on  solving  a  particular 
problem  and  they  were  able  to  measure  the  {uogress  of  the  development  activity 
with  more  precision  than  in  the  past  Since  the  imx:ess  relies  a  great  deal  on 
logical  thinking  as  opposed  to  programming  skill,  less  experienced  programmas 
are  able  to  take  a  bigga  share  of  the  development  burden. 
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6.  CcHiclusions 


The  most  important  conclusion  noted  by  this  effort,  even  in  its  preliminary  form,  is  that  the 
motivation  to  continue  to  use  Qeanroom  practices  and  PGPM  at  Picatinny  has  been  established. 
This  demonstration  effort  was  qwnsored  by  STARS  and  the  continued  e:^ort  is  being  sponsored 
by  the  AMCX!OM  LCSEC.  This  result  is  an  instance  of  the  STARS  program  fulfilling  its 
mission  by  being  the  catalyst  for  introducing  inprovemoits  to  the  software  engineering 
capabilities  in  the  DoD.  In  one  sense,  that  is  the  most  definitive  conclusion  of  this  effort;  the 
effort  is  to  be  expanded  across  the  entire  organization. 

In  addition  to  the  above  mentioned  conclusion  to  this  effort,  the  following  seven  conclusions  can 
be  drawn  based  on  the  current  status  of  the  MBC  and  I-COFT  projects. 

1.  It  is  possible  to  tranter  CSE  practices  to  project  teams  operating  within  a  SEJ  CMM  level 
1  organization. 

This  was  shown  by  the  fact  that  the  MBC  project  has  progressed  to  a  point  where  CSE  is  being 
successfully  applied.  This  result  shows  that  a  specific  maturity  rating  is  not  necessary  in  order 
to  benefit  fi-om  Cleamoom  Software  Engineering.  The  engineering  staff  also  enjoyed  using  the 
ideas,  and  all  were  interested  in  using  the  ideas  again.  Additionally,  nearly  all  were  interested 
in  supporting  and  participating  in  the  establishment  of  a  "Qeanroom  Competency  Center"  at  the 
Picatirmy  Arsenal. 

2.  SEI  CMM  level  1  organizations  can  realize  important  ben^ts  from  the  application  of  CSE. 

This  conclusion  is  supported  by  the  apparent  doubling  of  productivity  of  the  MBC  team. 
Although  it  is  too  early  to  make  predictions  about  quality,  the  MBC  team  is  excited  about  the 
proq)ect  of  the  upcoming  test  of  their  first  increment.  Thus,  any  result  achieved,  whether  it  be 
positive  or  negative,  will  be  viewed  by  the  MBC  team  as  the  mark  to  better  on  the  second 
inaement  of  this  project  The  incentive  and  motivation  for  continual  improvement  is  firmly  in 
place  among  MBC  team  members. 

3.  It  is  possible  to  tranter  PGPM  practices  to  project  teams  operating  within  a  SEI  CMM  level 
1  organization. 

This  was  shown  by  the  fact  that  PGPM  has  been  stux^essfully  and  enthusiastically  applied  in  the 
MBC  project  Once  again,  this  result  shows  that  a  specific  process  maturity  is  not  a  precondition 
to  the  successful  use  of  these  techniques. 

4.  SEI  CMM  level  I  organizations  can  realize  important  ben^ts  from  the  application  of  PGPM. 

Two  important  observations  from  the  MBC  project  are  that  (1)  PGPM  has  aided  the  learning 
process  and  helped  ease  the  transfer  and  application  of  the  CSE  technology  and  (2)  following  a 
well  (tefined  process  significantly  improves  team  productivi^. 
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5.  The  return  on  investment  at  Picatinny  cannot  be  definitively  calculated,  but  indications  are 
that  there  is  a  significant  return  on  investment. 


Since  neither  project  is  yet  complete,  a  preliminary  estimate  of  return  on  investment  can  only  be 
based  on  estimates  from  the  information  currently  available.  A  detailed  analysis  appears  at  the 
end  of  Appendix  A.  The  resulting  productivity  gains  and  return  on  investment  appear  in  Table 
V. 


Table  V;  Projected  Productivity  Change  and  Return  on  Investment  (ROI)  for  MBC 


MBC  Project  (Not 

Including  Training) 

MBC  Project  (Induding 
Training) 

Productivity  change  based  on 
Picatinny  basdine 

+  66% 

+  41% 

Productivity  change  based  tm 
ncatinny  b^dine  of 
Government  staffed  projects 
only 

+  120% 

+  87% 

ROI  based  on  Picatinny 
baseline 

3.31  :  1 

2.43  :  1 

ROI  based  mi  Picatinny 
baseline  of  Government  staffed 
projects  mily 

6.09 :  1 

5.14 :  1 

If  these  assotions  are  correct,  one  must  also  realize  that  productivity  will  increase  with  the  later 
increments  because  specifications  are  complete  for  the  entire  system.  Once  again,  the  final 
calculation  of  return  on  investment  awaits  project  conq)letion. 

6.  A  Computer  Aided  Process  Support  System  (P^)  facilitates  technology  transfer. 

Automating  the  non-creative  tasks  of  a  new  technology,  such  as  file  access  and  simple  process 
flow  facilitates  the  adoption  of  the  new  technology.  This  was  true  even  for  a  system  with 
limitations  known  and  subsequently  observed  in  CEPA.  CEPA’s  successor  system  (being 
developed  for  deployment  on  the  Air  Force  demonstration  project)  applies  many  of  the  lessons 
learned  from  observing  CEPA  use  at  Picatinny. 
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7.  Based  on  this  demonstration  we  now  believe  that  a  technology  transfer  program  to  support 
individual  projects  at  a  level  1  organization  should  consist  of  the  following  five  components:  (1 ) 
formal  CSE  training,  (2)  training  in  PGPM,  (2)  the  availability  of  engineering  handbooks,  (4) 
the  use  of  a  PSS  (e.g.,  CEP  A  and  its  successor),  and  (5)  the  availability  of  qualified  coaching. 


The  combination  of  technology  transfer  components  created  a  series  of  successes  at  Picatinny; 
including  productivity  gains,  expected  quality  gains,  and  the  increased  motivation  of  the 
engineering  staff. 

The  MBC  project  has  realized  the  most  significant  gains  from  the  CSE  ideas.  Once  the  learning 
curve  had  been  conquered  to  the  point  of  useful  application,  initial  successes  in  creating  the 
Black  Box  specification  served  to  cement  commitment  to  CSE. 

The  resulting  conclusions  from  the  overall  evaluation  are  preliminary  because  the  demonstration 
projects  are  still  in  their  early  stages.  However,  the  origind  hypothesis  that  Qeanroom  improves 
the  effectiveness  of  the  software  support  activities  at  Picatinny  looks  very  promising.  Indeed, 
management  and  staff  agree  that  morale  and  motivation  is  extremely  high,  that  teamwork  is  now 
the  normal  mode  of  operation,  and  that  people  are  excited  about  the  software  process  being 
established  and  are  motivated  to  produce  high  quality  products. 

A  good  technical  road  map  is  in  place  at  Picatinny;  the  technical  personnel  are  developing  the 
skills  that  appear  to  show  significant  gains  in  productivity.  Even  more  promising  is  the  fact  that 
these  gains  were  made  with  minimal  exposure  to  CSE.  Future  gains  are  likely  to  be  of  greater 
magnitude  as  projects  are  carried  out  by  experienced  teams  of  Cleanroom  engineers  well 
advanced  on  the  learning  curve. 
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7.  Recommendations 


Recommendations  based  on  this  wcnk  are  made  in  three  areas  as  follows: 

1)  Actions  tt>  support  continued  process  in^rovement  at  Picatinny. 

2)  Actions  Peterson  Air  Force  Base  can  undertake  to  facilitate  their  demonstration  project. 

3)  Actions  ARPA  can  undertake  to  siq)port  continued  learning  for  how  to  best  utilize 
these  new  software  engineering  practices  to  improve  the  level  of  software  produced  by 
DoD  organizations. 

Recommendations  for  Picatinny 

Our  recommendations  for  the  future  of  software  practices  at  Picatiimy  are  that  CSE  should 
continue  to  be  main-streamed  into  the  process  and  technology  for  software  engineering.  The 
employment  of  CSE  practices  has  demonstrated  improvements  in  attitude,  competence  and 
process  control,  which  are  the  building  blocks  of  process  maturity  and  ongoing  quality 
inq)rovement  Picatinny  has  already  taken  the  necessary  steps  to  continue  IBM  STARS  team 
support  for  Cleanroom  Software  Engineering  and  Process-Guided  Project  Management 

Our  specific  recommendations  for  Picatinny  are  as  follows: 

1.  Picatinny  needs  to  establish  a  Cleanroom  competency  group. 

This  group  should  be  responsible  for  continuous  review  and  study  of  the  application  of 
Qeanroom  and  PGPM  at  Picatinny.  This  group’s  charter  should  be  to  internalize  and  refine  CSE 
and  PGPM  to  the  software  practices  at  Picatinny. 

2.  The  experienced  members  of  the  MBC  project  should  establish  a  training  program  that 
supports  actional  CSE  projects  at  Picatinny. 

The  technology  transfer  package  developed  by  SET  and  IBM  can  be  used  by  the  trained  staff  to 
support  additional  projects.  The  MBC  team  is  fortunate  to  have  talented  team  members  who  can 
carry  this  out 

3.  A  program  needs  to  be  instituted  to  upgrade  Picatinny’ s  SEI CMM  rating  to  at  least  a  level 

3. 

Picatinny  is  incentivized  to  move  up  to  a  level  3,  due  to  DoD  assertions  that  at  some  time  in  the 
future,  software  support  activities  that  ftmction  below  a  level  3  will  be  closed.  In  effect 
Picatinny,  like  all  other  software  support  activities,  must  evolve  just  to  stay  in  business.  This 
step  requires  a  concentrated  management  effort  and  commitment  to  ongoing  education  and 
training  of  Picatinny  personnel.  Also,  new  CSE  projects  should  be  instituted  so  that  additional 
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experience  can  be  gained  in  applying  disciplined  software  engineering  and  give  the  opportunity 
for  measurement  and  improvement  It  is  our  hypothecs  that  organization  process  levels  can  best 
be  inq>roved  by  a  combination  of  bottom-up  improvements  resulting  from  project  level  work  and 
top-down  work  with  the  organization’s  management  The  details  of  how  to  do  this  at  Picatinny 
will  need  to  be  developed. 

4.  Organize  the  tranter  of  the  Process  Support  System  (PSS)  being  developed  for  the  Air  Force 
demonstration  project  to  Picatinny. 

The  understood  and  observed  shortcomings  in  CEPA  are  primarily  addressed  by  the  PSS  being 
developed  by  the  IBM  STARS  team  for  use  on  the  Air  Force  Denranstration  Project  at  Peterson 
Air  Force  Base.  The  major  source  that  provided  input  to  the  specification  process  for  the  PSS 
was  the  experience  gained  by  using  CEPA  at  Picatinny.  As  a  result  many  of  the  improvements 
desired  by  Picatinny  are  being  developed  as  a  part  of  the  PSS.  The  PSS  needs  to  be  delivered 
to  Picatinny,  as  well  as  to  Peterson  Air  Force  Base.  As  a  result  Picatinny  will  receive  their 
desired  functionality  and  will  help  provide  a  second  test  bed  for  the  PSS. 

Recommendations  for  Peterscm  Air  Force  Base 

From  the  perspective  of  Peterson  Air  Force  Base,  the  Picatinny  experience  can  be  viewed  as  a 
"dry  run"  for  the  technology  transfer  process  tiiat  will  occur  in  Colorado,  starting  in  the  fall  of 
1993.  Based  on  the  experiences  of  the  technology  transfer  effort  a  number  of  recommendations 
can  be  made.  It  should  be  noted  that  the  experiences  and  recommendations  are  independent  of 
project  size;  the  fact  tiiat  the  project  at  Peterson  Air  Force  Base  (PAFB)  will  be  bigger  only 
accentuates  the  importance  of  these  recommendations.  The  recommendations  are  as  follows: 

1)  PATO  and  the  IBM  STARS  ream  need  to  recognize  the  three  aspects  of 
technology  transfer  (Organizational,  Technological  and  Process)  and  consider  them 
when  developing  the  Technology  Transfer  Plan  for  the  STARS  demonstration. 

2)  The  Technology  Transfer  Plan  should  entail  the  five  parts  of  a  technology  transfer 
program  that  were  recognized  at  Picatinny.  These  parts  are: 

a)  formal  Qeanroom  Software  Engineering  training, 

b)  training  in  Process  Guided  Project  Management, 

c)  the  availability  of  engineering  handbooks, 

d)  the  use  of  a  Process  Support  Sysrem  (e.g.,  CEPA  and  its  successor),  and 

e)  the  availability  of  qualified  coaching 

3)  PAFB  and  the  IBM  STARS  team  need  to  jointly  develop  a  technology  transfer 
plan  that  clearly  defines: 

a)  the  technologies  to  be  transferred, 

b)  the  objectives  of  the  technology  transfer  effort, 

c)  the  means  by  which  the  technology  transfer  effort  will  be  measured, 

d)  a  description  of  the  organization  and  project  on  which  the  technology  transfer 
effort  will  be  conducted,  and 

e)  the  detailed  plan  of  how  the  technology  transfer  will  be  conducted. 
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This  technology  transfer  plan  will  eliminate  die  ambiguities  and  misconceptions 
that  may  exist,  by  clearly  and  con^letely  defining  what  the  technology  transfer 
is  and  how  it  will  be  conducted. 

4)  PAFB  will  need  to  create  Statonents  of  Work  (SOW’s)  for  their  contractors  that 
have  contained  the  content  of  die  Technology  Tranter  Plan.  Specifically,  the 
SOW’s  must  define  the  contmt  of  deliverables  and  the  manner  in  which  the  woric 
will  be  conducted,  in  oida  to  ensure  that  the  Technology  Transfo*  effort  is 
conducted. 

5)  A  Process  Support  System  (PSS)  should  be  used  to  accelerate  technology 
adoption. 

6)  The  importance  of  the  process  management  tool  on  the  PSS  must  be  recognized. 
That  tool  will  give  project  teams  the  ability  to  conduct  effective  process-guided 
project  management  It  should  be  noted  that  the  shortcomings  of  fee  PSS  system 
used  at  Picatinny  are  being  precluded  from  fee  PSS  being  developed  to  support 
fee  Peterson  Air  Force  Base  demonstration. 

The  Picatinny  results  should  provide  reassurance  to  fee  Peterson  Air  Force  Base  and  ARPA 
managers  that  three  of  fee  technologies  (Qeanroom  software  engineering,  process-guided  project 
nuioagement  automated  process  support)  fee  IBM  STARS  team  is  recommending  for 
deimmstration  and  refinement  at  Peterson  have  a  high  probability  of  providing  fee  desired  impact 
The  Picatinny  experience  has  contributed  to  mitigating  fee  risk  of  fee  larger  Peterson 
demonstration,  especially  since  fee  weaknesses  identified  at  Hcatinny  in  fee  area  of  automated 
process  support  have  been  remedied  for  fee  Peterson  demonstration. 

Reconmiendatioiis  for  ARPA 

This  project  provides  a  roadmap  for  developing  fee  means  to  support  process  improvement  in 
DoD  software  support  sites.  It  is  recommended  thiti  ARPA  support  additional  technology  transfer 
efforts  along  fee  lines  taken  at  Picatinny,  in  order  to  leverage  fee  results  obtained.  Such  work 
will  produce  tangible  returns  for  fee  DoD  as  well  as  develop  a  substantial  body  of  knowledge 
about  fee  best  way  to  improve  software  engineoing  processes.  The  DoD  and  industry  have  a 
desire  to  have  their  software  engineering  organizations  operating  at  a  high  SEI  CMM  level  of 
noaturi^.  It  is  observed  tiiat  fee  prevailing  belief  is  feat  process  maturity  improvements  must  be 
made  top  down  ova*  a  long  period  of  time.  A  working  hypofeesis  is  feat  a  combined  approach 
of  workiag  from  fee  bottom-up  and  fee  top-down  can  greatly  reduce  fee  time  and  effort  required 
to  accomplish  a  substantial  improvement  in  SEI  CMM  maturity  levels.  A  program  to  test  this 
hypothesis  and  develop  fee  means  to  implement  it  would  lead  to  substantial  gains  for  DoD  as 
well  as  industry  in  general. 


1DS2  -  Final  Evaluation  Report 


CDRL  05503-004  Page  26 


Appendix  A  -  Metrics 


Deiinitioii  and  Nfeasuranent  of  Demonstration  and  Control  Group  Metrics 

The  Technology  Transfer  Plan  (submitted  to  the  government  as  CDRL  0SS01-(X)1  under  task  ID- 
S2  on  May  6,  1992)  defines  die  set  of  metrics  to  be  comqmted  on  die  demonstration  and  control 
group  projects.  This  t^ipendix  is  reproduced  and  enhanced  from  that  document  These  evaluation 
metrics  address  productivity,  quality  and  cycle  time  over  accumulations  of  the  projects. 
Productivity  is  measured  in  terms  of  effort  based  on  reported  time  by  the  technical  staff.  (Quality 
is  judged  via  observed  failures/defects  in  every  phase  of  the  develo|mient  and  dqployment 
process.  Cycle  time  is  computed  based  on  the  duration  of  die  project  A  summary  of  the 
fomiulas  used  to  compute  each  appears  below.  Rrier  to  the  above  CDRL  for  a  more  complete 
analysis  of  the  fcnmulas  and  additional  refinonents  of  some  of  the  calculations. 

In  ordo'  to  create  an  appropriate  unit  of  ouqiut  for  a  project  for  the  above  metrics,  a  standard  for 
conqiuting  lines  of  code  (LOC)  has  been  established.  A  project  is  denoted  as  i. 

L(X![i]a(Nuia  of  New  LOC[/]>KNuin.  LCX^  in  Modified  Components[i]) 

The  lines  of  code  in  modified  components  is  necessary  due  to  the  maintenance  nature  of 
Picatinny  mhancement  activity.  The  reason  for  including  an  entire  modified  component  is 
because  in  Qeanroom,  the  entire  conqioiient  is  analyzed  and  understood  in  order  to  isolate  and 
institute  modifications.  It  is  often  the  case  that  LOC  are  rqported  in  thousand  units,  denoted 
KLOC  or  thousand  lines  of  code. 

Upon  initial  work  at  Picatinny,  it  was  realized  that  tiiere  was  a  necessity  to  determine  how  to 
factor  in  changed  code,  since  little  code  was  developed  new.  As  a  result,  the  approach  used  at 
NASAAjoddard  Space  Flight  Center  was  used  to  measure  the  dfort  required  to  develop  changed 
code.  The  means  used  to  measure  changed  code  is  to  determine  Um  number  of  lines  in  a  changed 
conqionent,  and  mult^ly  that  by  .2.  The  reasoning  behind  this  is  that  even  though  a  small  part 
of  the  coirqionent  may  need  to  be  changed,  the  full  component  must  be  sufficiently  read  and 
niMterstood  in  order  to  be  properly  modified.  As  a  result,  productivity  was  measurod  by  the 
following  formula. 

LC)C[i]=(Num.  of  New  LC)C[/])-K.2*(Num.  LOC  in  Modified  Components!/])) 

Productivity  is  the  rate  or  output  per  unit  time.  In  the  case  of  Picatinny,  time  is  measured  in  staff 
months. 


PRODU(:TIVITY=(LC)C[i])/(Technical  Staff  Months!/]) 

(Quality  of  a  software  product  is  temporal,  making  it  necessary  to  measure  both  the  pre- 
deployment  and  post-deployment  quality. 
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PRE-QUALrrY-=(Num.  of  Pre'dq>loyinent  Failiires[/])/KLOC[i]) 

PC)ST>QUALITY-=(Num.  of  Post-deployment  Failures(i])/KLOC[/]) 

Cycle  time  measures  project  duration  in  terms  of  total  calendar  time  elapsed  from  project 
inception  until  project  termination.  Of  course,  one  needs  precise  d^nitions  of  when  to  start  and 
stop  the  clock  to  conqiutB  cycle  time. 

Return  on  Investment  is  a  ratio.  The  ratio  is  derived  from  a  relationsh^  between  the  benefit  of 
the  use  of  the  factorCs)  of  the  investment  and  the  cost  of  the  investment  itself,  that  is  the  cost  to 
acquire  the  factor.  Both  the  benefit  and  the  investment  are  calculated  in  dollars.  For  the 
Picatinny  ocpeiiment,  one  of  the  results  the  STARS  program  desires  is  the  return  on  investment 
of  acquiring  the  Qeanroom  technologies  at  Software  Support  Activities.  Stated  in  terms  of  a 
formula: 


ROI=BENEFIT  FROM  INVESTMENT/COST  OF  INVESTMENT 

This  measure  of  effectivoiess  is  a  projection  based  on  a  function  of  the  relative  changes  in  toms 
of  jmxluctivity,  quality  and  cycle  time  from  tiie  expoimental  projects,  and  a  projection  of  the 
cost  to  acquire  the  experimental  factor.  The  cost  of  investment  which  represents  the  cost  of  the 
acquisition  of  the  frictors.  Restating  the  formula: 

f(CHANGE  IN  PRODUCTIVITY,  CHANGE 
IN  QUALITY.  CHANGE  IN  CYCLE  TIME) 

(12)  ROI  =  COST  TO  ACQUIRE  CLEANROOM  TECHNOLOGIES 

Data  used  for  the  measurements  comes  from  both  off-line  and  on-line  sources.  Off-line  measures 
are  tiiose  that  take  place  on  completed  projects.  Thus,  data  is  gadwred,  validated  and  its  context 
re-created  so  that  a  basis  for  comparison  can  be  established.  On-line  measures  are  those  that  take 
place  during  project  execution.  On-line  measures  are  significantly  easier  to  gather  and  validate 
since  one  can  prescribe  the  measures  to  be  made  and  organize  the  data  to  be  collected  to  make 
the  measurement 

Actual  Calculation  of  Projected  Return  on  Investment  Based  (ni  the  Currait  State  of  the 
MBC  Project 

The  following  analysis  uses  only  savings  and  costs  from  the  MBC  project  to  support  the 
technology  transfer  to  both  projects.  At  this  time,  specifications  for  the  entire  MBC  system  are 
complete  and  development  for  Ae  first  increment  of  code  is  nearly  done.  Based  on  data  currently 
available  from  discussions  with  MBC  project  staff,  it  is  believed  tiiat  total  project  effort  through 
first  increment  certification  will  total  22.5  staff  months  (plus  4  staff  months  of  training  -  two 
weeks  time  for  8  people).  The  MBC  project  staff  believe  the  first  increment  will  result  in  4500 
lines  of  Ada  code.  Staff  productivity  is  based  on  160  hour  staff  months,  with  govonment 


ID52  -  Final  Evaluation  Report 


CDRL  05503-004  Page  28 
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empk^ees  costing  $61  per  staff  tour.  The  total  amount  of  IBM  STARS  team  direct  project 
si^KXt  (training  and  coaching  fcff  MBQ  was  $43,723. 

The  results  of  the  calculations  appear  in  Table  V. 
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